Papers with modality fusion
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies study sentiment and emotion separately and do not fully exploit the complementary knowledge behind the two. |
| Approach: | They propose a multimodal sentiment knowledge-sharing framework that unifies MSA and ERC tasks from features, labels, and models. |
| Outcome: | The proposed framework achieves consistent improvements on four public benchmark datasets on MOSI, MOSEI, MELD, and IEMOCAP. |
CrisisTS: Coupling Social Media Textual Data and Meteorological Time Series for Urgency Classification (2025.acl-long)
Copied to clipboard
| Challenge: | Existing studies on fusion of texts and tabular-based time series to improve performance of NLP applications have focused on coupling texts with tabular time series. |
| Approach: | They propose a multimodal and multilingual dataset for urgency classification that allows for temporal and location alignment even in the absence of location mention in the text. |
| Outcome: | The proposed dataset outperforms text-only models in many applications while ensuring model generalizability. |
Capturing Latent Modal Association For Multimodal Entity Alignment (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for multimodal entity alignment overlook the quality of input modality embeddings during modality interaction, amplifying noise propagation while suppressing discriminative feature representations. |
| Approach: | They propose a model for capturing latent modal association for multimodal entity alignment using a self-attention mechanism to enhance salient information while attenuating noise within individual modality embeddings. |
| Outcome: | The proposed model achieves an absolute 3.1% higher Hits@1 score than the sota method. |
Lost in Embeddings: Information Loss in Vision–Language Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Experiments reveal connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40–60% post-projection, correlating with degradation in retrieval performance. |
| Approach: | They propose two approaches to examine and quantify information loss by analyzing latent representation space. |
| Outcome: | The proposed model improves retrieval performance by analyzing changes in k-nearest neighbor relationships between image representations before and after projection. |
Latent Distribution Decouple for Uncertain-Aware Multimodal Multi-label Emotion Recognition (2025.findings-acl)
Copied to clipboard
Jingwang Huang, Jiang Zhong, Qin Lei, Gaojinpeng Gaojinpeng, Ymyang Ymyang, Sirui Wang, PeiguangLi PeiguangLi, Kaiwen Wei
| Challenge: | Existing studies focus on improving fusion strategies and modeling modality-to-label dependencies, but they overlook the impact of aleatoric uncertainty, which is inherent noise in multimodal data. |
| Approach: | They propose a latent emotional distribution decomposition with uncertainty perception framework to model aleatoric uncertainty in multimodal data. |
| Outcome: | The proposed framework achieves state-of-the-art performance on the CMU-MOSEI and M3ED datasets, highlighting the importance of uncertainty modeling in MMER. |
Cross-lingual Matryoshka Representation Learning across Speech and Text (2026.findings-acl)
Copied to clipboard
| Challenge: | Speakers of under-represented languages face language barriers and modality barriers . we train a bilingual speech-text embedding model for French-Wolof . |
| Approach: | They train a bilingual speech-text Matryoshka embedding model that enables efficient retrieval of French text from Wolof speech queries. |
| Outcome: | The proposed model can retrieve French text from Wolof speech queries without expensive ASR-translation pipelines. |